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This  Advanced  Development  effort  was  conducted  in  support  of  Project  Z0108-PN 
(ZPN07),  Education  and  Training  Development,  under  the  sponsorship  of  the  Chief 
of  Naval  Operations  (OP-099) . This  is  the  fourth  in  a series  of  reports  relating 
to  Subproject  Z0108-PN. 24,  Personnel  Readiness  Training.  The  first  report,  NPRDC 
Special  Report  75-8,  A Personnel  Readiness  Training  Program;  Initial  Project 
Developments,  provided  an  overview  of  the  project.  Subsequent  reports,  NPRDC 
Technical  Report  77-4,  A Personnel  Readiness  Training  Program;  Operation 
of  the  AN/BQR-20A.  and  NPRDC  Technical  Report  77-19,  A Personnel  Readiness 
Training  Program;  Maintenance  of  the  Missile  Test  and  Readiness  Equipment 
(MTRE  Mk  7 MOD  2) . described  the  program  application  for  the  submarine  Sonar 
Technician  and  Missile  Technician,  respectively.  A final  report  will  summarize 
findings  and  conclusions  across  all  three  applications. 

In  this  application.  Boiler  Technicians  used  training  materials  covering 
basic  mechanical  skills  and  knowledges.  These  materials  were  developed  by 
the  Propulsion  Engineering  School,  Service  School  Command,  Great  Lakes  and  adapted 
for  shipboard  use  by  the  Naval  Education  and  Training  Support  Center,  Pacific, 

San  Diego.  The  Boiler  Technicians  also  used  materials  covering  the  Engineering 
Operational  Sequencing  System.  These  were  developed  under  a work  agreement  with 
the  Naval  Instructional  Technology  Development  Center,  San  Diego,  who  awarded 
a subcontract  to  Manpower  Development  Services,  Evanston,  Illinois. 

Appreciation  is  expressed  to  the  following  commands  and  their  staffs: 

• Commander  Fleet  Training  Center,  San  Diego,  for  making  available  the 
facilities  of  the  Individualized  Learning  Center. 

• Commanding  Officer,  Recruit  Training  Command,  Naval  Training  Center, 

San  Diego,  for  help  in  the  validation  of  the  Engineering  Operational 
Sequencing  System  Test. 

• Director,  Propulsion  Engineering  School,  Service  School  Command,  Great 
Lakes,  Illinois,  for  help  in  the  validation  of  the  Basic  Mechanical  Procedures 
Test. 

• Commander  Naval  Surface  Force,  Pacific  Fleet;  Commander  Cruiser  Destroyer 
Groups  3 and  5;  and  Commander  Destroyer  Squadrons  7,  9,  17,  and  31  for  allowing 
their  ships  to  participate  in  the  program  on  a volunteer  basis. 

Special  appreciation  is  expressed  to  the  Commanding  Officers,  Engineering 
Officers,  and  Boiler  Technicians  of  the  13  cruisers  and  destroyers  who  volunteered 
to  participate  in  the  testing  program. 

Finally,  the  substantial  and  valuable  assistance  of  the  following  persons 
is  gratefully  acknowledged:  Ray  E.  Main  for  developing  the  behavioral  objectives 

and  test  items  of  the  Engineering  Operational  Sequencing  System  Module,  Carolyn 
McLandrlch  for  writing  a computer  program  used  in  scoring  the  test,  Robert  C, 
Panell  for  analyzing  the  data,  and  BTC  Danny  L.  Bowers  and  MMC  Jon  Hall  for 
writing  test  items. 


J.  J.  CLARKIN 
Commanding  Officer 
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SUMMARY 


Problem 


The  Personnel  Readiness  Training  Program  Is  concerned  with  the  feasibility 
of  using  a diagnostic  testlng/shlpboard  training  system  to  improve  the  readiness 
levels  of  Fleet  personnel.  In  such  a system,  performance-oriented  tests  are 
used  to  diagnose  deficiencies  in  job  performance  and  shipboard  self-instructional 
materials  are  Individually  prescribed  to  correct  deficiencies  revealed  by  the 
diagnostic  tests.  To  obtain  Information  on  how  and  where  this  type  of  system 
might  work,  testing  and  training  programs  were  developed  for  three  applications: 
(1)  the  submarine  Sonar  Technician  (ST)  operating  the  AN/BQR-20A,  (2)  the 
submarine  Missile  Technician  (MT)  maintaining  the  Missile  Test  and  Readiness 
Equipment  (MTRE  Mk  7 Mod  2),  and  (3)  the  Boiler  Technician  (BT)  operating  and 
maintaining  the  1200  PSI  Steam  Propulsion  Plant. 

Purpose 

The  purpose  of  the  effort  described  in  this  report  was  to  determine  whether 
the  testing  and  training  programs  developed  for  the  BT  application  were  Instru- 
mental in  Improving  essential  basic  skills  and  knowledges  that  support  the 
operation  and  maintenance  of  the  1200  PSI  Steam  Propulsion  Plant. 

Approach 

Boiler  Technicians  from  12  cruisers  and  destroyers  equipped  with  1200  PSI 
Steam  Propulsion  Plants  were  given  diagnostic  pretests  and,  with  the  exception 
of  one  crew,  retested  3 to  6 months  later.  Two  multiple-choice  tests  were 
given:  one  was  concerned  with  basic  skills  and  knowledges  required  to  support 

plant  maintenance,  and  the  other  with  basic  skills  and  knowledges  required  to 
support  plant  operation.  Both  tests  were  based  on  job  performance  requirements. 
Three  experimental  groups,  each  consisting  of  the  BTs  from  four  ships,  were 
used  In  the  evaluation:  a Control  Group,  a Diagnostic  Feedback  Group,  and  a 

Diagnostic  Feedback  + Training  Group.  Following  the  pretests,  members  of  the 
Control  Group  were  given  feedback  on  their  performance  In  terms  of  an  overall 
percentage  score.  Members  of  the  Diagnostic  Feedback  Group  were  provided  with 
a list  of  their  Individual  areas  of  weakness.  Members  of  the  Diagnostic  Feedback 
+ Training  Group  were  provided  with  the  same  type  of  Information  as  the  Diagnostic 
Feedback  Group,  but  they  were  also  assigned  specific  remedial  training  materials 
covering  their  Individual  areas  of  weakness. 

Findings 

Diagnostic  testing  was  successful  In  detecting  deficiencies  of  Fleet 
personnel  on  skills  and  knowledges  related  to  the  operation  and  maintenance 
of  the  1200  PSI  Steam  Propulsion  Plant.  In  the  pretest  sample,  only  17  of  the 
305  Boiler  Technicians  tested  showed  no  weaknesses.  Providing  the  Boiler 
Technicians  with  feedback  resulted  In  a very  small,  but  statistically  signifi- 
cant, Improvement  In  performance.  However,  there  was  little  effect  of  remedial 
training  because  only  21  percent  of  the  Boiler  Technicians  completed  their 
assigned  training  materials.  Secondary  analyses  suggested  that,  had  the  assigned 
training  been  completed.  It  would  have  Improved  performance.  One  of  the  reasons 
the  training  was  not  fully  completed  was  the  extensive  time  demands  placed  on 
fireroom  personnel.  Another  reason  the  training  was  not  fully  completed  may 
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have  been  the  lack  of  adequate  study  facilities  on  board  ship.  The  tvo  ships 
that  were  asked  to  study  the  materials  In  shorebased  classrooms  completed  the 
most  training  (61  percent  of  that  assigned) . Not  only  were  these  men  removed 
from  the  work  demands  and  distractions  of  the  shipboard  environment,  but  they 
%iere  also  provided  with  a place  to  study. 

Conclusions 

Diagnostic  testing  was  successful  In  detecting  deficiencies  of  Fleet 
personnel  on  written  Items  related  to  basic  skills  and  knowledges  essential  to 
the  operation  and  maintenance  of  the  1200  PSI  Steam  Propulsion  Plant.  The  data 
suggest  that,  had  the  assigned  training  been  completed.  It  would  have  Improved 
performance.  Feedback  on  diagnosed  weaknesses  had  a statistically  significant 
effect  but  little  practical  meaning.  It  would  appear  that  basic  skills  and 
knowledges  can  be  better  acquired  through  a formalized  Instructional  program 
than  solely  through  on-the-job  experience. 

In  a shipboard  training  program  for  propulsion  engineering  personnel,  the 
availability  of  study  time  appears  to  be  a key  factor,  although  an  adverse  study 
environment  may  also  play  a role.  Virtually  no  assigned  training  was  accom- 
plished on  board  ship,  while  a substantial  amount  was  completed  at  shore  facilities. 

Recommendations 

To  Improve  the  proficiency  level  of  the  BT  and  other  propulsion  engineering 
personnel  responsible  for  operating  and  maintaining  the  1200  PSI  Steam  Propulsion 
Plant,  It  Is  recommended  that: 

1.  The  training  package  supporting  plant  operation  (l.e.,  self  Instruction 
on  the  Engineering  Operational  Sequencing  System)  be  added  to  the  training 
package  supporting  plant  maintenance  currently  Installed  at  the  Individualized 
Learning  Centers  of  the  Fleet  Training  Centers  at  San  Diego  and  Norfolk. 


f . 


2.  Performance  deficiencies  be  Identified  by  administering  the  diagnostic 
test  package  to  flreroom  and  englneroom  personnel.  Remedial  training  materials  i 

should  be  assigned  by  means  of  the  procedures  developed  as  part  of  this  study. 
Arrangements  should  then  be  made  to  allow  the  use  of  the  Individualized  Learning 
Centers  on  a day-to-day  basis  so  that  required  remedial  training  can  be  completed 
with  minimal  disruption  of  ongoing  shipboard  activity.  i 


On  the  basis  of  the  findings  of  this  study,  the  use  of  Individualized, 
self-instructional  material  on  board  ship  Is  not  recommended  unless  sufficient 
study  time  and  an  appropriate  study  environment  can  be  made  available. 
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INTRODUCTION 


Problem 


The  Personnel  Readiness  Training  Program  is  concerned  with  the  feasibility 
of  using  a diagnostic  testing/shipboard  training  system  to  improve  the  readi- 
ness levels  of  Fleet  personnel.  In  such  a system,  performance-oriented  tests 
are  used  to  diagnose  deficiencies  in  job  performance  and  shipboard  self-instruc- 
tional materials  are  individually  prescribed  to  correct  deficiencies  revealed 
by  the  diagnostic  tests. 

Background 


The  degree  to  which  critical  job  skills  can  be  Improved  through  a system  of 
diagnostic  testing  and  shipboard  training  may  depend  on  the  rating  and/or  the 
type  of  task  involved.  Therefore,  to  obtain  information  on  how  and  where  the 
system  might  work,  testing  and  training  programs  were  developed  for  three 
applications:  (1)  the  submarine  Sonar  Technician  (ST)  operating  the  AN/BQR-20A, 

(2)  the  submarine  Missile  Technician  (MT)  maintaining  the  Missile  Test  and 
Readiness  Equipment  (MTRE  Mk  7 Mod  2),  and  (3)  the  Boiler  Technician  (BT)  operating 
and  maintaining  the  1200  PSI  Steam  Propulsion  Plant.  These  ratings  and  equipments 
were  chosen  for  program  evaluation  because  they  were  critical  to  the  missions 
of  their  ships  and  they  contained  a sufficient  variety  of  operator  and  maintenance 
tasks  to  permit  the  results  to  be  generalized  to  other  areas.  Additionally, 
there  were  indications  that  performance  deficiencies  might  be  present  in  these 
areas. 


The  first  report  in  this  series  (Laabs,  Main,  Abrams,  & Steinemann,  Note  1) 
described  the  general  approach  and  how  it  was  being  applied  in  the  three  areas. 
Subsequent  reports  (Laabs,  Panell,  & Pickering,  1977;  Winchell,  Panell,  & 
Pickering,  1976)  described  the  Personnel  Readiness  Training  Program  for  the 
MT  and  ST.  A final  report  will  summarize  findings  and  conclusions  across  all 
three  applications. 


Purpose 

The  1200  PSI  Steam  Propulsion  Plant  is  a high  pressure  steam  plant  that 
has  been  plagued  with  problems  of  unreliability  caused,  in  part,  by  training 
deficiencies.  Routine  maintenance  of  this  steam  plant  requires  that  the  BT 
be  well  grounded  in  basic  mechanical  skills  and  knowledges;  proper  operation 
requires  a thorough  knowledge  of  the  Engineering  Operational  Sequencing  System 
(EOSS) , a system  of  step-by-step  operating  procedures.  Deficiencies  in  these 
essential  skills  and  knowledges  reduce  personnel  readiness  that,  in  turn,  reduces 
operational  readiness,  but  such  deficiencies  can  be  overcome  by  training.  It 
is  important  to  distinguish  these  deficiencies  from  other  factors,  such  as  lack 
of  spare  parts,  poor  equipment  design,  and  personnel  shortages,  which  also  reduce 
operational  readiness.  The  problems  caused  by  the  latter  factors  cannot  be 
overcome  by  training. 


The  purpose  of  the  present  effort  was  to  determine  whether  the  testing  and 
training  programs  developed  for  the  BT  were  Instrumental  in  improving  skills 
and  knowledges  that  support  the  operation  and  maintenance  of  the  1200  PSI  Steam 
Propulsion  Plant. 

i 


1 


METHOD 


Experimental  Design 

Boiler  Technician  crews  from  12  Pacific  Fleet  cruisers^  and  destroyers 
equipped  with  1200  PSI  Steam  Propulsion  Plants  participated  In  the  evaluation 
of  the  BT  application  of  the  Personnel  Readiness  Training  Program.  As  depicted 
in  the  experimental  design,  shown  in  Table  1,  three  experimental  groups,  each 
consisting  of  BT  crews  from  four  ships,  were  used  in  the  evaluation:  a Control 

Group,  a Diagnostic  Feedback  Group,  and  a Diagnostic  Feedback  + Training  Group. 
Before  assignment  to  experimental  groups,  sets  of  three  ships  were  matched  on 
the  basis  of  ship  type,  expected  time  at  sea,  and  BT  crew  size.  From  these 
four  sets  of  three  ships,  one  ship  was  assigned  to  each  experimental  group. 
Members  of  all  three  experimental  groups  were  to  be  given  a diagnostic  pretest 
and  retested  approximately  3 to  6 months  later.  What  happened  to  the  BT  crews 
during  the  interim  depended  upon  the  group  to  which  they  were  assigned  (see 
Table  1).  Due  to  the  adverse  environment  of  the  BT  shipboard  spaces,  two  of 
the  crews  who  were  assigned  remedial  training  materials  were  asked  to  use 
study  facilities  ashore,  and  the  other  two  to  use  spaces  on  board  ship. 


Table  1 

Experimental  Design  Used  in  the  BT  Application  of  the 
Personnel  Readiness  Training  Program 


Group  Experimental  Conditions 


Control 

Pretest  with  non- 
specific feedback 

Usual  Operations 

Pos  tires  t 

Diagnostic 

Feedback 

Pretest  with  specific 
feedback  on  weaknesses 

Usual  Operations 

Posttest 

Diagnostic 

Training 

Feedback  + 

Pretest  with  specific 
feedback  on  weaknesses 
and  remedial  training 
assigned 

Usual  Operations 
Special  Training 
Packages  On  Board 
or  Ashore 

Posttest 

^Three  ships  in  the  sample  requested  that,  in  addition  to  their  BTs,  their 
Machinist  Mates  (MMs)  be  tested.  Because  the  performance  of  the  MMs  was 
very  simlllar  to  that  of  the  BTs,  they  were  included  in  the  final  sample. 

The  MMs  constituted  only  17  percent  of  the  final  sample  and,  for  ease  of 
exposition,  the  sample  is  referred  to  as  being  comprised  of  BT  crews. 
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Remedial  Training  Materials 

Basic  Mechanical  Skills  and  Knowled>;es 

In  all  applications  of  the  Personnel  Readiness  Training  Program, 
existing  materials  (such  as  task  analyses,  tests,  or  training  materials)  were 
used  whenever  possible.  In  the  BT  maintenance  area,  previously  developed 
training  materials  could  be  used.  The  core  curriculum  of  the  Propulsion 
Engineering  School,  Service  School  Command,  Great  Lakes,  covers  the  basic  mechan- 
ical skills  and  knowledges  essential  for  the  routine  maintenance  of  propulsion 
engineering  plants.  This  core  curriculum  is  not  only  completed  by  recruits 
who  are  designated  to  become  BTs,  but  also  by  those  designated  to  become 
Machinist  Mates  (MMs)  and  Enginemen  (ENs).  It  consists  of  modularized,  self- 
paced  instruction,  which  makes  it  an  ideal  source  for  individually  prescribed 
remedial  Instruction.  Because  the  curriculum  had  been  adapted  for  shipboard 
use  by  the  Naval  Education  and  Training  Support  Center,  Pacific,  it  could  be 
readily  incorporated  into  the  BT  application.  It  also  had  been  installed  at 
the  Individualized  Learning  Center,  Fleet  Training  Center,  San  Diego,  which 
enabled  a comparison  to  be  made  between  training  conducted  ashore  and  on  board 
ship.  This  is  especially  important  to  the  BT  application  because  training  at 
a shore  installation  is  an  obvious  way  of  overcoming  the  adverse  environment 
of  the  BT  shipboard  spaces. 

A total  of  13  core  curriculum  modules,  ranging  in  topic  from  use  of 
hand  tools  to  oil  pollution,  support  the  maintenance  of  the  steam  propulsion 
plant.  In  addition,  a 14th  module  on  the  Planned  Maintenance  System  (PMS) , 
was  adapted  for  shipboard  use  from  another  part  of  the  Propulsion  Engineering 
School  curriculum.  It  was  included  because  PMS  documents  were  used  as  part 
of  the  diagnostic  test.  As  shown  in  Table  2,  which  lists  the  basic  skills  and 
knowledges  modules  that  were  used  in  the  BT  application,  the  estimated  time 
to  complete  a module  ranged  from  1 hour  for  the  shortest  module  (Module  IS' 
to  6 hours  for  the  longest  modules  (Modules  4 and  6) . 

Engineering  Operational  Sequencing  System  (BOSS) 

The  Engineering  Operational  Sequencing  System  (BOSS)  is  a set  of 
documents  that  provide  detailed  operating  procedures  for  the  1200  PSI  Steam 
Propulsion  Plant,  this  system,  which  is  relatively  new  and  complex,  contains 
not  only  step-by-step  procedures  for  operating  the  steam  plant,  but  also  plant 
and  space  status  diagrams,  preferred  alignment  diagrams,  troubleshooting  charts, 
and  casualty  control  procedures.  If  it  is  understood  and  used  correctly,  it 
results  in  the  safe  steaming  of  the  plant  by  all  propulsion  engineering  personnel. 

Under  a work  agreement  with  the  Naval  Instructional  Technology 
Development  Center,  San  Diego,  instructional  material  on  BOSS  and  a 10-minute 
motivational  film  on  the  value  of  BOSS  were  produced.  The  self-instructional 
materials,  which  were  developed  under  subcontract  to  Manpower  Development 
Services,  Bvanston,  Illinois,  included  the  following  lessons:  (1)  Introduction 

to  BOSS,  (2)  Engineering  Operational  Procedures,  (3)  Engineering  Operational 
Casualty  Control,  and  (4)  Supervising  and  Managing  BOSS.  To  complete  these 
lessons,  the  BT  was  required  to  work  with  reproductions  of  BOSS  documents  bound 
in  a sample  BOSS  book.  The  estimated  time  to  complete  each  lesson  was  2-1/2 
hours. 
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Table  2 


Estimated  Amount  of  Study  Time  Required  for  Completion  of 
Modules  Covering  Basic  Skills  and  Knowledges 


Module 

Title 

Study  Time  (Hrs.) 

1 

Metal  Fasteners,  Hand  Tools 

2-1/2 

2 

Pipe,  Tubing,  Fittings 

2 

3 

Packing,  Gaskets,  Insulation 

2-1/2 

4 

Valves 

6 

5 

Bearings,  Lubrication 

3-1/2 

6 

Pumps 

6 

7 

Precision  Measuring  Instruments, 
Technical  Manuals 

5 

8 

Heat  Properties,  Heat  Exchangers 

3 

9 

Indicating  Devices 

3 

10 

Turbines,  Couplings,  Gears 

4 

11 

Strainers  (Lesson  1) 

2 

12 

Low  Pressure  Air  System  and  Compressor 

4 

13 

Oil  Pollution 

1 

14 

Planned  Maintenance  System 

1-1/2 

Just  the 
oil  purifiers 

first  lesson  was  used  because  the  remaining 
and  are  only  completed  by  Machinist  Mates. 

lessons  concern 
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Diagnostic  Tests 

The  MT  and  ST  applications  of  the  Personnel  Readiness  Training  Program 
(Laabs,  Panell,  & Pickering,  1977;  Wlnchell,  Panell,  & Pickering,  1976)  relied 
primarily  upon  hands-on  performance  tests  to  diagnose  deficiencies.  It  was 
planned  to  follow  the  same  approach  for  BT  operator  tasks,  while  using  a job- 
related  written  test  for  BT  maintenance  tasks  (Laabs,  Main,  Abrams,  & Steinemann, 
Note  1).  Therefore,  a hands-on  performance  test,  which  required  a ship  to 
operate  its  steam  plant  in  an  auxiliary  mode  and  to  set  a new  flreroom  watch 
every  hour  until  all  watch  teams  had  been  observed,  was  constructed  for  adminis- 
tration on  board  ships  in  port.  Unfortunately,  permission  could  not  b"  obtained 
to  administer  this  hands-on  test  to  the  12  ships  in  the  sample  because  of  exten- 
sive demands  already  placed  on  the  ships.  In  fact,  permission  could  not  even 
be  obtained  to  try  out  the  test  on  a single  ship.  Consequently,  the  BT  applica- 
tion had  to  be  altered  to  provide  for  a written,  performance-oriented  test  on 
steam  plant  operation. 

Basic  Mechanical  Procedures  Test 


This  diagnostic  test  contained  104  multiple-choice  questions.^  To 
ensure  that  the  test  would  be  job-related,  descriptions  were  provided  of  hypo- 
thetical job  situations,  which  were  derived  from  known  job  requirements.  Under 
each  such  description,  questions  were  asked  that  required  the  demonstration  of 
both  skills  (e.g.,  reading  a simulated  measuring  Instrument  such  as  a micrometer) 
and  knowledges  (e.g.,  identifying  components  of  equipments  and  their  functions) 
that  support  the  job  and  that  are  covered  in  the  basic  modules.  Answering  the 
questions  involved  many  actions  that  simulated  those  occurring  on  the  job, 

(e.g.,  identifying  tools,  reading  measurement  devices,  recognizing  equipment 
components,  and  interpreting  charts,  tables,  and  diagrams). 

The  test  was  developed  in  two  stages.  In  the  first  stage,  a pool  of 
186  test  items  was  administered  to  200  BTs  who  were  students  at  the  Propulsion 
Engineering  School,  Great  Lakes.  Half  of  them,  representing  the  pretest  group, 
were  just  entering  the  course,  and  the  other  half,  representing  the  posttest 
group,  had  just  completed  the  basic  modules  listed  in  Table  2.  A random  sample 
of  25  students  from  each  group  was  designated  as  a cross-validation  sample. 

Thus,  an  item  analysis  was  completed  using  the  responses  of  the  remaining  75 
students  in  each  group.  Those  items  for  each  module  (for  a total  of  94)  that 
maximally  discriminated  between  the  two  groups  were  selected  for  inclusion  in 
the  diagnostic  test.  Next,  a cutoff  score  (l.e.,  pass/fall  criterion)  for  each 
module  was  established  based  on  the  performance  of  the  group  that  had  completed 
the  core  curriculum.  These  cutoff  scores  were  used  to  classify  the  students  in 
the  cross-validation  sample.  That  is,  if  a student  attained  a score  that  was 
equal  to  or  above  the  cutoff  score,  he  would  be  classified  as  being  like  a post- 
test group  member.  If  he  attained  a score  below  the  cutoff  score,  he  would  be 
classified  as  being  like  a pretest  group  member.  When  this  classification  was 
compared  to  actual  group  membership,  there  was  substantial  agreement  for  all 
but  two  modules. 


^Six  additional  questions  are  available  on  the  part  of  Module  11  concerning 
oil  purifiers. 
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In  the  second  stage,  which  was  conducted  to  refine  the  test,  a pool  of 
127  items  (the  94  items  previously  selected  plus  33  new  items)  was  administered 
to  150  students  at  the  Propulsion  Engineering  School,  none  of  whom  had  partici- 
pated in  the  first  stage.  As  with  the  first  group,  half  of  the  students  were 
just  entering  the  course  and  half  had  just  completed  the  core  curriculum.  Again, 
25  students  from  each  group  were  selected  at  random  as  a cross-validation 
sample,  and  an  item  analysis  was  conducted  on  the  responses  of  the  remaining 
50  students  in  each  group.  Those  items  for  each  module  (for  a total  of  110) 
that  maximally  discriminated  between  the  two  groups  were  selected  for  inclusion 
in  the  final  test.  As  before,  cutoff  scores  for  each  module  were  used  to  classify 
members  of  the  cross-validation  sample  and  there  was  substantial  agreement 
for  all  modules.  These  cutoff  scores  were  used  to  assign  modules  to  members 
of  the  Diagnostic  Feedback  + Training  Group.  For  example,  if  the  cutoff  score 
for  a certain  module  was  8 and  a member  of  the  group  scored  5,  he  would  be 
considered  deficient  in  the  topic  covered  by  that  module  and  be  assigned  to 
complete  it.  A more  detailed  description  of  the  item  analysis  and  criterion 
determination  procedures  can  be  found  in  Laabs  and  Panell,  Note  2. 

EOSS  Test 


This  diagnostic  test,  which  contained  28  multiple-choice  questions, 
used  the  same  format  as  the  Basic  Mechanical  Procedures  Test.  That  is,  it  was 
both  job-related  and  performance-oriented,  because  it  involved  reading  and  inter- 
preting EOSS  documents  that  were  reproduced  in  the  test  booklet. 

The  method  used  to  develop  the  EOSS  test  was  similar  to  that  used 
to  develop  the  Basic  Mechanical  Procedures  Test;  however,  there  were  not 
sufficient  personnel  available  to  perform  extensive  cross  validation.  That  is, 
a pool  of  test  items  was  administered  to  27  BTs  from  a destroyer  who  had  completed 
the  EOSS  module  and  33  students  who  had  just  finished  the  Fireman  Apprentice 
Course  at  the  Naval  Training  Center,  San  Diego. ^ The  performance  of  the  two 
groups  was  compared,  and  those  questions  for  each  lesson  that  maximally  discrimi- 
nated between  them  were  used  to  make  up  the  test.  Lesson  cutoff  scores 
were  determined  on  the  basis  of  the  performance  of  the  BTs  who  had  completed 
the  module.  These  cutoff  scores  were  used  to  assign  EOSS  lessons  to  the 
Diagnostic  Feedback  + Training  Group. 

Procedures 


Pretest 


A total  ot  305  BTs  participated  in  the  pretesting,  with  95  BTs  assigned 
to  the  Control  Group;  98,  to  the  Diagnostic  Feedback  Group;  and  112,  to  the 
Diagnostic  Feedback  + Training  Group.  To  minimize  demands  on  the  participating 
ships,  testing  was  conducted  in  a dockside  van  at  San  Diego  and  Long  Beach  by 
members  of  the  research  team.  The  BTs  were  tested  in  groups  of  10. 


^The  2-week  Fireman  Apprentice  Course  is  attended,  immediately  after  recruit 
training,  by  those  BTs  not  sent  to  A School.  Only  one  40-mlnute  lecture  describing 
EOSS  is  given  during  the  course. 


Before  the  pretests  were  administered,  the  BTs  were  Informed  that 
the  testing  program  was  experimental  in  nature  and  that  they  would  be  retested 
during  the  next  3 to  6 months.  They  were  not  told,  however,  that  there  were 
three  different  groups  Involved.  Depending  upon  the  group  to  which  they  were 
assigned,  each  BT  crew  was  told  the  type  of  feedback  Information  they  would 
receive  following  testing.  For  example,  members  of  the  Diagnostic  Feedback 
-t-  Training  Group  were  told  they  would  be  given  an  outline  of  their  deficiencies 
and  an  Individualized  prescription  for  remedial  training.  In  addition,  members 
of  this  group  were  shown  the  10-mlnute  motivational  film  related  to  the  use 
of  BOSS.  The  remaining  groups  were  shown  the  film  at  postbest. 

Control  Group.  Following  the  pretest,  members  of  this  group  were  given 
feedback  on  their  test  performance  In  terms  of  an  overall  percentage  score. 

For  example,  a BT  was  told  that  his  scores  were  65  percent  on  the  Basic  Mechan- 
ical Procedures  Test  and  56  percent  on  the  BOSS  Test.  Members  of  this  group 
were  not  provided  with  any  Information  concerning  their  specific  deficiencies 
and  they  were  not  given  suggestions  or  directions  as  to  how  they  might  correct 
their  deficiencies. 

Diagnostic  Feedback  Group.  Following  the  pretest,  members  of  this 
group  were  provided  with  an  outline  Indicating  their  Individual  weaknesses. 

If  a member  of  this  group  did  not  meet  the  cutoff  score  established  for  assign- 
ment of  a module  to  members  of  the  Diagnostic  Feedback  + Training  Group,  the 
topic  of  the  module  was  listed  as  an  area  of  weakness.  For  example.  If  a BT 
missed  4 or  more  questions  from  the  set  of  12  questions  keyed  to  the  module 
on  pumps,  he  was  informed  he  was  weak  In  that  area.  As  with  the  Control  Group, 
members  of  this  group  were  not  provided  with  any  training  materials  or  Informa- 
tion as  to  how  their  deficiencies  might  be  corrected. 

Diagnostic  Feedback  + Training  Group.  Following  the  pretest,  members 
of  this  group  were  given  the  same  type  of  feedback  as  was  given  to  those  In 
the  Diagnostic  Feedback  Group;  that  Is,  a list  of  Individual  weaknesses.  In 
addition,  they  were  assigned  specific  modules  covering  their  Individual  weak- 
nesses. For  example.  If  a BT  missed  7 or  more  questions  from  the  set  of  17 
questions  keyed  to  the  module  on  valves,  he  was  Informed  he  was  weak  In  that 
area  and  was  assigned  to  complete  the  module  on  valves  at  times  convenient 
for  him  during  the  next  3 to  6 months.  So  that  the  modules  could  be  reused, 
each  BT  was  supplied  with  a notebook  containing  a list  of  his  weaknesses  and 
assignments,  and  providing  space  for  him  to  Insert  his  answers.  A master  list 
of  modules  assigned  to  each  BT  was  provided  to  the  Engineering  Officer  for 
monitoring  training  and  maintaining  a record  of  conq>leted  modules.  To  evaluate 
the  effect  of  the  study  environment,  two  of  the  ships  In  this  group  were  asked 
to  have  the  assigned  training  materials  completed  on  board;  and  the  remaining 
two,  to  have  them  completed  ashore.  For  the  "ashore"  ships,  one  was  to  send 
Its  BTs,  on  a day-to-day  basis,  to  the  Individualized  Learning  Center,  Fleet 
Training  Center,  San  Diego;  and  the  other,  to  classrooms  available  at  the 
Long  Beach  Naval  Shipyard.  Just  when  the  training  was  to  be  accomplished  was 
left  up  to  the  individual  ship. 

Post test 


The  posttest  administered  to  aJ 1 groups  was  the  same  as  the  pretest. 
However,  one  control  ship,  with  a crew  size  of  42,  could  not  be  posttested. 

In  addition,  90  BTs  were  unavailable  for  posttesting  due  to  discharge,  transfer, 
or  leave,  thereby  reducing  the  Control  Group  to  28;  the  Diagnostic  Feedback 
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Group,  to  76;  and  the  Diagnostic  Feedback  + Training  Group,  to  74.  The 
distribution  of  pay  grades  and  the  percentage  of  the  final  sample  each  represents 
are  shown  In  Table  3. 


Table  3 


Percentage  Distribution  of  Pay  Grades  in  the  Experimental  Groups 


1 

> 

L 

i 

i 

Pay  Grades 

Diagnostic 
Feedback  + 
Training  Group 
(N  - 74) 

Diagnostic 
Feedback 
Group 
(N  - 76) 

Control 
Group 
(N  - 28) 

Total 
Sample 
(N  - 178) 

[ 

i- 

E-3  and  below 

45 

45 

50 

46 

f 

E-4 

40 

42 

43 

42 

E-5 

15 

13 

7 

12 

RESULTS 


Basic  Mechanical  Procedures  Test 


Reliability  and  Validity 


Since  the  Basic  Mechanical  Procedures  Test  was  scored  with  respect 
to  predetermined  cutoff  points.  It  should  be  classified  as  a criterion-referenced 
test.  Therefore,  most  standard  measures  of  reliability  and  validity  developed 
for  norm-referenced  tests  do  not  apply  (Popham  & Husek,  1969).  The  procedures 
used  to  estimate  the  reliability  and  validity  of  the  Basic  Mechanical  Procedures 
Test  are  outlined  below  and  are  described  In  more  detail  In  Laabs  and  Panell, 

Note  2. 


In  a program  where  the  purpose  Is  to  diagnose  deficiencies  and  to 
assign  training  to  remedy  those  deficiencies,  test  reliability  can  be  viewed 
In  terms  of  the  consistency  of  the  diagnostic  decisions  made  at  two  different 
points  In  time.  Since  the  Control  Group  received  no  special  treatment  (l.e., 
no  specific  feedback  and/or  training  materials),  their  pre-  and  posttest  scores 
on  the  basic  modules  provided  data  to  estimate  test-retest  reliability.  Table 
4 shows  the  percent  of  agreement  In  diagnostic  decisions  based  on  these  test 
scores.  Even  though  3 to  6 months  passed  between  the  two  test  administrations, 
the  percent  agreement  in  diagnostic  decisions  for  the  Individual  modules  ranged 
from  a high  of  96  percent  to  a low  of  71  percent. 


The  Basic  Mechanical  Procedures  Test  has  both  high  face  and  content 
validity  because  the  test  questions  were  related  to  job  situations  that  the  BT 
is  riquired  to  do  routinely.  Job  experts  were  used  to  determine  what  information 
contained  In  the  modules  was  needed  for  a given  job  and  to  write  the  appropri- 
ate test  questions.  The  extensive  use  of  charts,  diagrams,  and  Illustrations 
in  presenting  each  job  situation  added  to  both  the  face  and  content  validity. 

A measure  of  validity  was  calculated  during  the  development  of  the 
test.  As  Indicated  previously,  for  the  items  selected  for  a module,  a cutoff 
score  was  established  on  the  basis  of  scores  attained  by  students  who  had 
completed  the  core  curriculum.  During  test  refinement,  the  cutoff  was  applied 
to  the  scores  of  a separate  cross-validation  sample  of  25  students  who  had 
completed  the  modules  and  25  who  had  not.  Members  of  the  cross-validation 
sample  who  achieved  the  cutoff  score  for  a module  were  classified  as  not  needing 
training;  and  those  who  did  not,  as  needing  training.  This  classification  was 
then  compared  to  actual  group  membership.  As  shown  In  Table  5,  the  percent 
agreement  between  actual  group  membership  and  group  membership  based  on  test 
scores  for  each  of  the  basic  modules  ranged  from  a high  of  92  percent  to  a low 
of  68  percent. 


Deficiencies  In  the  Basics 

Table  6 shows  that,  of  the  305  BTs  pretested,  about  10  percent  showed 
no  deficiencies  (l.e.,  30  BTs  met  or  exceeded  the  cutoff  score  on  all  14  modules), 
2 percent  failed  to  reach  the  cutoff  score  on  any  of  the  14  modules,  and  about 
50  percent  needed  to  complete  five  or  more  modules.  (In  the  final  sample  of 
187  BTs,  about  7 percent  showed  no  deficiencies.) 


Table  4 


Reliability  of  Diagnostic  Decisions  for  Two  Administrations 
of  the  Basic  Mechanical  Procedures  Test — Control  Group 

(N  - 28) 


Percent 

Module 

Title 

Agreement 

1 

Metal  Fasteners,  Hand  Tools 

89 

2 

Pipe,  Tubing,  Fittings 

86 

3 

Packing,  Gaskets,  Insulation 

79 

4 

Valves 

86 

5 

Bearings,  Lubrication 

71 

6 

Pumps 

82 

7 

Precision  Measuring  Instruments, 
Technical  Manuals 

75 

8 

Heat  Properties,  Heat  Exchangers 

71 

9 

Indicating  Devices 

71 

10 

Turbines,  Couplings,  Gears 

71 

11 

Strainers  (Lesson  1) 

96 

12 

Low  Pressure  Air  System  and  Compressor 

82 

13 

Oil  Pollution 

71 

14 

Planned  Maintenance  System 

75 
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Table  5 


Group  Membership  Determined  by  the  Basic  Mechanical  Procedures  Test 
Cross-validated  Against  Actual  Group  Membership — Test  Development  Group 

(N  - 50) 


^ Percent 

Module 

Title 

Agreement 

1 

Metal  Fasteners,  Hand  Tools 

88 

2 

Pipe,  Tubing,  Fittings 

78 

3 

Packing,  Gaskets,  Insulation 

68 

4 

Valves 

86 

5 

Bearings,  Lubrication 

74 

6 

Pumps 

92 

7 

Precision  Measuring  Instruments, 
Technical  Manuals 

86 

8 

Heat  Properties,  Heat  Exchangers 

72 

9 

Indicating  Devices 

78 

10 

Turbines,  Couplings,  Gears 

80 

11 

Strainers  (Lesson  1) 

88 

12 

Low  Pressure  Air  System  and  Compressor 

88 

13 

Oil  Pollution 

86 

14 

Planned  Maintenance  System 

90 

\ 
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Table  6 


Percentage  of  BTs  in  the  Pretest  Sample  Who  Were  Diagnosed 


as  Needing  the  Indicated 

Number  of  Basic 

Modules 

4 

/' 

Experimental  Groupi 

s 

} 

Number  of 

Diagnostic 

Diagnostic 

[ 

Modules 

Feedback  + Training 

Feedback 

Control 

Total 

V 

Needed 

(N  = 112) 

(N  = 98) 

(N  = 95) 

(N  = 305) 

\ 

14 

0 

4 

1 

2 

■ 

13 

4 

3 

1 

3 
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A separate  analysis  for  each  module  was  hindered  because  the  great 
majority  of  the  BTs  In  the  Diagnostic  Feedback  + Training  Group  did  not  complete 
their  assigned  training  (only  21  percent  of  these  BTs  completed  the  Basic 
Mechanical  Procedures  modules  they  were  assigned).  Thus,  the  main  analysis 
Involved  the  comparison  of  the  total  score  achieved  over  all  modules  on  the  Basic 
Mechanical  Procedures  Test  for  each  experimental  group.  Only  those  mend>ers 
of  each  group  who  failed  to  meet  the  cutoff  scores  for  one  or  more  of  the  modules 
were  Included  In  the  analysis.  The  overall  percent  correct  at  pre-  and  posttest 
for  the  three  experimental  groups  Is  shown  In  Table  7.  This  total  score  analysis 
had  the  effect  of  combining  small  changes  In  each  of  the  14  areas  that  might 
not  be  statistically  significant  when  analyzed  separately.  An  analysis  of  vari- 
ance with  repeated  measures  on  the  pre-/posttest  variable  Indicated  that  (1)  the 
overall  average  Improvement  of  3 percent  was  significant  (1,  160)  = 14.92, 

2_  ^ .001),  and  (2)  there  were  differential  score  Increases  for  the  groups  from 
pre-  to  posttest  (JF  (2,  160)  - 6.22,  £ < .005).  Further  analyses  revealed  small 
but  significantly  greater  score  Increases  for  both  the  Diagnostic  Feedback  + 
Training  Group  and  the  Diagnostic  Feedback  Group  than  for  the  Control  Group 
(F^  (1,  160)  ■ 35.35,  £ < .001,  £ (1,  160)  = 10.72,  £ < .001;  respectively). 

The  very  small  additional  effect  of  training  over  that  of  feedback  was  not 
surprising  In  view  of  the  extremely  low  completion  rate.  Separate  analyses 
were  performed  on  the  data  for  the  Individual  modules  using  only  those  members 
of  each  group  who  failed  to  meet  the  cutoff  for  each  module.  In  general,  the 
results  of  these  analyses  confirmed  that  the  changes  due  to  training  and/or 
feedback  were  too  small  to  be  statistically  significant  when  observed  for  the 
separate  modules. 


Table  7 


Overall  Percent  Correct  on  the  Basic  Mechanical  Procedures  Test 
at  Pre-  and  Posttest  for  Three  Experimental  Groups 


Groups 

Condition 

Diagnostic 
Feedback  + Training 
(N  - 70) 

Diagnostic 
Feedback 
(N  - 70) 

Control 
(N  - 23) 

Pretest 

70 

72 

73 

Posttest 

76 

75 

72 

Note.  BTs  who  exceeded  cutoff  on  all  modules  are  not  Included. 
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BOSS  Test 


Reliability  and  Validity 

The  multiple-choice  BOSS  Test  was  designed  to  diagnose  deficiencies 
In  the  use  and  knowledge  of  BOSS  so  that  training  could  be  assigned  to  remedy 
those  deficiencies.  Using  the  same  procedure  followed  with  the  Basic  Mechanical 
Procedures  Test,  the  reliability  of  the  BOSS  Test  was  estimated  In  terms  of  the 
consistency  of  the  diagnostic  decisions.  For  the  BOSS  Introduction  lesson,  there 
was  79  percent  agreement  In  diagnostic  decisions  from  pre-  to  posttest;  for  the 
Bnglneerlng  Operational  Procedure  lesson,  64  percent;  for  the  Engineering 
Operational  Casualty  Control  lesson,  61  percent;  and  for  the  BOSS  Supervision 
and  Management  lesson,  61  percent. 

The  BOSS  Test  was  developed  In  a manner  similar  to  that  used  with  the 
Basic  Mechanical  Procedures  Test.  It  has  both  high  face  and  content  validity 
because  of  extensive  use  of  reproduced  BOSS  material  and  because  the  questions 
were  based  upon  actual  Job  requirements.  To  answer  test  questions,  the  BTs  had 
to  refer  to  BOSS  diagrams,  charts,  and  tables  that  would  be  used  on  the  job. 
Unfortunately,  there  was  not  a sufficient  number  of  BTs  available  during  the 
development  of  this  test  to  cross- validate  It,  as  was  done  In  the  development 
of  the  Basic  Mechanical  Procedures  Test. 

Use  and  Knowledge  of  BOSS 

Table  8 shows  that,  of  the  305  BTs  in  the  pretest  sample,  about  65 
percent  failed  to  reach  the  cutoff  score  on  one  or  more  of  the  BOSS  lessons 
and  about  35  percent  showed  no  deficiencies.  (In  the  final  sample  of  187  BTs, 
about  28  percent  showed  no  deficiencies.) 


Table  8 

Percentage  of  BTs  In  the  Pretest  Sample  Who  Were  Diagnosed 
as  Needing  the  Indicated  Number  of  BOSS  Lessons 


Number  of 

Lessons 

Needr>d 

Experimental  Groups 

Diagnostic 
Feedback  + 
Training 
(N  - 112) 

Diagnostic 
Feedback 
(N  - 98) 

Control 
(N  - 95) 

Total 
(N  - 305) 

4 

5 

3 

2 

4 

3 

18 

14 

14 

15 

2 

21 

23 

26 

24 

1 

20 

28 

18 

22 

0 

36 

31 

40 

35 

L.I 
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Skills  and  Knowledges  Deficiencies 

The  pretest  results  showed  that  there  were  many  deficiencies  In  answering 
sets  of  performance-oriented,  written  items  related  to  the  operation  and 
maintenance  of  the  1200  PSI  Steam  Propulsion  Plant.  The  seriousness  of  the 
problem  is  indicated  by  the  fact  that  the  test  only  covered  basic  skills  and 
knowledges  that  were  considered  to  be  essential  for  proper  operation  and  main- 
tenance of  the  steam  propulsion  plant.  Only  6 percent  of  the  pretest  sample, 
which  was  composed  of  Pay  Grades  E-5  and  below,  showed  no  deficiencies. 

An  examination  of  the  effect  of  previous  school  experience  on  performance 
on  the  Basic  Mechanical  Procedures  Test  revealed  that  the  BTs  who  had  attended 
A School  failed  fewer  modules  (4.1)  than  those  who  had  not  (8.3).  For  those 
BTs  who  had  attended  A School,  it  made  little  difference  whether  or  not  they 
had  attended  before  or  after  the  school  instituted  the  current  self-paced 
instructional  program.  Further,  there  was  no  effect  of  school  experience  on 
performance  on  the  BOSS  Test.  This  might  be  expected  because  the  EOSS  Module 
was  developed  especially  for  the  present  program  and  not  adapted  from  the  A 
School  curriculum  as  were  the  other  14  modules. 

The  results  from  the  11  BT  crews  retested  showed  that  the  performance  of 
all  three  experimental  groups  improved  a very  small,  but  significant,  amount 
from  pre-  to  posttest  on  their  total  scores  for  the  Basic  Mechanical  Procedures 
Test.  However,  both  groups  that  received  feedback  showed  more  improvement  than 
the  Control  Group.  Although  this  differential  improvement  was  statistically 
significant,  it  should  be  recognized  that  it  was  so  small  as  to  have  little 
practical  meaning.  In  general,  there  was  no  differential  group  improvement 
when  the  15  modules  (including  the  EOSS  Module)  were  analyzed  separately. 

Lack  of  a Training  Effect 

Since  the  great  majority  of  BTs  in  the  Diagnostic  Feedback  + Training 
Group  did  not  complete  their  assigned  modules,  it  is  not  surprising  that  there 
was  little  evidence  of  improvement  due  to  training.  Because  of  the  extremely 
low  completion  rate  for  all  15  of  the  modules  (only  21  percent  of  the  BTs  com- 
pleted their  assigned  modules) , it  was  not  clear  whether  better  posttest  scores 
would  have  been  attained  if  the  modules  assigned  had  been  completed.  Therefore, 
the  scores  of  those  BTs  who  had  completed  a given  module  were  compared  with 
those  of  similar  BTs  from  the  other  two  groups.  That  is,  members  of  the  Control 
Group  and  Diagnostic  Feedback  Group  who  would  have  been  assigned  that  same  module 
were  selected  to  form  a yoked  subsample  for  that  module.  Unfortunately,  yoked 
subsamples  of  even  marginal  size  could  not  be  formed  for  most  of  the  modules. 

Only  the  data  from  the  six  modules  for  which  nine  or  more  BTs  completed  their 
assigned  training  were  analyzed.  Table  10  shows  the  percentage  score  at  pre- 
and  posttest  for  the  six  yoked  subsamples  under  each  experimental  condition. 

In  four  of  the  six  cases,  there  was  a differential  increase  in  score  from  pre- 
to  posttest,  as  indicated  by  the  significant  statistical  interactions.  In  each 
of  these  cases,  the  score  increase  for  the  Diagnostic  Feedback  + Training  Group 
was  significantly  greater  than  that  of  the  other  two  groups,  as  indicated  by 
the  test  of  simple  main  effects  shown  in  Table  11.  In  addition,  the  differ- 
ential score  increase  approached  significance  for  one  of  the  remaining  two 
modules.  Module  5 (£  (2,  32)  * 3.0,  needed  3.30  for  £ < .05),  with  the  Diagnostic 
Feedback  + Training  Group  again  showing 
groups. 


Table  11 
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While  the  above  comparisons  are  based  on  very  small  subsamples,  they 
do  suggest,  that  had  training  been  completed.  It  would  have  Improved  performance. 
Additional  support  on  this  point  was  obtained  from  an  analysis  of  the  Basic 
Mechanical  Procedures  Test  scores  for  the  BT  crew  that  had  completed  the  greatest 
percentage  of  assigned  modules.  This  crew's  completion  rate  was  79  percent  as 
compared  to  an  average  rate  of  15  percent  for  the  other  three  training  crews. 

Its  Improvement  rate  was  23  percent  as  compared  to  6 percent  for  the  other  three 
training  crews. 

On  the  Role  of  Study  Time  and  Environment 

There  are  two  evident  reasons  for  the  nonuse  of  training  materials:  (1) 

the  lack  of  time  to  complete  training  on  board  ship,  and  (2)  the  poor  study 
environment  on  board  ship.  One  ship  in  the  Diagnostic  Feedback  4-  Training  Group 
reported  that,  for  a 2-month  period,  it  spent  approximately  85  percent  of  Its 
time  at  sea  with  engineering  personnel  standing  port  and  starboard  watches  (l.e., 
a continuous  sequence  of  6 hours  on  watch  followed  by  6 hours  off).  As  might 
be  expected,  this  ship  completed  the  least  amount  of  assigned  training  (only 
1 percent  of  that  assigned).  The  other  ship  that  was  asked  to  study  Instructional 
materials  on  board  ship  completed  only  10  percent  of  the  training  assigned. 

To  add  to  the  burden  of  normal  operations,  both  of  these  ships  were  preparing 
for  Inspections  by  the  Mobile  Training  Team  and  the  Propulsion  Examination  Board. 
In  addition  to  the  heavy  work  load,  the  adverse  conditions  in  the  BT  work  spaces 
are  not  conducive  to  the  study  of  training  materials. 

The  two  ships  asked  to  study  Instructional  materials  on  shore  completed 
61  percent  of  the  assigned  training  while  the  other  two  ships  completed  only 
5 percent.  The  completion  rate  of  the  ship  that  used  shipyard  classrooms 
(46  percent  of  the  training  assigned)  would  have  been  much  higher  if  It  had  not 
been  for  an  overzealous  training  supervisor  who  decided  that  his  crew  should 
complete  all  the  modules.  Savings  In  training  time,  which  should  have  resulted 
from  the  Individualized  remediation,  were  not  realized  because  everyone  was 
started  on  Module  1 and  proceeded  sequentially.  As  might  be  expected,  most 
of  the  earlier  modules  assigned  were  completed,  but  very  few  of  the  later  ones. 
However,  the  total  amount  of  time  spent  on  studying  the  modules  was  approximately 
that  required  to  complete  Just  those  modules  diagnosed  as  needed.  The  best 
completion  rate  (79  percent  of  the  training  assigned)  was  logged  by  the  ship 
that  was  asked  to  study  Instructional  materials  at  the  Individualized  Learning 
Center,  Fleet  Training  Center,  San  Diego. 

Supplementary  Material  Costs 

A large  portion  of  the  present  program's  training  material  Is  currently 
available  for  use  either  on  board  ship  or  at  the  Individualized  Learning  Centers 
of  the  Fleet  Training  Centers  at  San  Diego  and  Norfolk. Therefore,  the  cost 
associated  with  Implementing  the  present  testlng-and-trainlng  program  is  limited 
mainly  to  obtaining  supplementary  materials  as  outlined  on  the  following  page. 


''Additional  copies  of  the  13  basic  modules  can  be  obtained  from  the  Naval 
Publications  and  Forms  Center,  Philadelphia  (NAVEDTRA  7500,  Stock  Number  0502- 
LP0375000). 
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I In  lots  of  300,  the  cost  of  adding  the  Basic  Mechanical  Procedures  Test  to 

f the  13  basic  modules  Is  estimated  at  less  than  $1.00  per  set.  The  additional 

F cost  of  expanding  this  set  of  modules  to  Include  Module  14  and  15  (Including 

i the  BOSS  Test)  Is  estimated  at  approximately  $5.00  per  set.  The  cost  per  set 

^ would  Increase  if  the  modules  and  diagnostic  tests  were  only  integrated  into 

tthe  program  available  at  the  Individualized  Learning  Centers  of  the  Fleet  Training 
Centers  at  San  Diego  and  Norfolk.  In  lots  of  50,  the  cost  per  set  Is  estimated 
at  approximately  $25.00.  Table  12  provides  a breakdown  of  estimated  cost  by 
v ; material  and  lot  size. 


Table  12 

Estimated  Cost  of  Testing  and  Training  Materials 


Estimated  Cost 
Per  Copy 

Materials 

Components 

50  Lot 

300  Lot 

Module  14  (Planned 
Maintenance  System) 

52  Printed  Pages 
2 Foldouts 

$ 3.74 

$1.25 

Module  15  (Engineering 
Operational  Sequencing 
System — EOSS) 

120  Printed  Pages 
4 Separate  Lessons 

$10.40 

$3.48 

Sample  EOSS  Book 

129  Printed  Pages 
20  Binders 
1 Binder 

$ 8.00 

$2.67 

EOSS  Test 

16  Printed  Pages 
1 Assignment  Guide 

$ 1.56 

$ .25 

Basic  Mechanical 
Procedures  Test 

42  Printed  Pages 
1 Assignment  Guide 

$ .76 

$ .52 
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Diagnostic  testing  was  successful  In  detecting  deficiencies  of  Fleet 
personnel  on  written  Items  related  to  basic  skills  and  knowledges  essential  to 
the  operation  and  maintenance  of  the  1200  PSI  Steam  Propulsion  Plant.  The  data 
suggest  that,  had  the  assigned  training  been  completed.  It  would  have  Improved 
performance.  Feedback  on  diagnosed  weaknesses  had  a statistically  significant 
effect  but  little  practical  meaning.  It  would  appear  that  basic  skills  and 
knowledges  can  be  better  acquired  through  a formalized  Instructional  program 
than  solely  through  on-the-job  experience. 

In  a shipboard  training  program  for  propulsion  engineering  personnel,  the 
availability  of  study  time  appears  to  be  a key  factor,  although  an  adverse  study 
environment  may  also  play  a role.  Virtually  no  assigned  training  was  accom- 
plished on  board  ship,  while  a substantial  amount  was  completed  at  shore 
facilities. 
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RECOMMENDATIONS 

To  improve  the  proficiency  level  of  the  BT  and  other  propulsion  engineering 
personnel  responsible  for  operating  and  maintaining  the  1200  PSI  Steam  Propulsion 
Plant,  it  is  recommended  that: 

1.  The  training  package  supporting  plant  operation  (l.e,,  self  Instruction 
on  the  Engineering  Operational  Sequencing  System)  be  added  to  the  training 
package  supporting  plant  maintenance  currently  installed  at  the  Individualized 
Learning  Centers  of  the  Fleet  Training  Centers  at  San  Diego  and  Norfolk. 

2.  Performance  deficiencies  be  identified  by  administering  the  diagnostic 
test  package  to  fireroom  and  englneroom  personnel.  Remedial  training  materials 
should  be  assigned  by  means  of  the  procedures  developed  as  part  of  this  study. 
Arrangements  should  then  be  made  to  allow  the  use  of  the  Individualized  Learning 
Centers  on  a day-to-day  basis  so  that  required  remedial  training  can  be  completed 
with  minimal  disruption  of  ongoing  shipboard  activity. 

On  the  basis  of  the  findings  of  this  report,  the  use  of  individualized, 
self-instructional  material  on  board  ship  is  not  recommended,  unless  sufficient 
study  time  and  an  appropriate  study  environment  can  be  made  available. 
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